analyze: deduplicated size of a set of archives, #5741 - #9971
Merged
ThomasWaldmann merged 2 commits intoJul 30, 2026
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #9971 +/- ##
==========================================
+ Coverage 85.77% 85.83% +0.05%
==========================================
Files 95 95
Lines 16869 17034 +165
Branches 2582 2607 +25
==========================================
+ Hits 14470 14621 +151
- Misses 1665 1673 +8
- Partials 734 740 +6 ☔ View full report in Codecov by Harness. |
ThomasWaldmann
commented
Jul 30, 2026
ThomasWaldmann
commented
Jul 30, 2026
`borg analyze` now reports how much space a set of archives occupies and how much deleting it would actually free. All figures come as source (uncompressed source data) and stored (compressed, as stored in the repository) size, plus the compression factor relating the two. For the considered (matching) set of archives: - deduplicated size of the set: the summed size of the union of chunks the set references (chunks shared within the set counted once), - exclusive size of the set: the summed size of chunks referenced only by the set and by no other archive, i.e. what deleting the whole set would free, - unreferenced chunks: chunks no non-deleted archive references, i.e. what `borg compact` could free in the current state of the repository. Following ThomasWaldmann's proposal in borgbackup#5741, chunks are flagged as referenced by the considered set (C) and/or by the rest of the archives (R); the exclusive size is sum(size(x) for x in C - R). Without an archive filter the considered set is all archives, so everything referenced is trivially exclusive to it - the report is then labelled as covering the whole repository and that line is left out. `--by-name` decomposes the whole repository by archive name instead: one row per name with what is exclusive to it, one row for the chunks shared by 2+ names and one for the unreferenced ones. Every chunk is counted in exactly one row, so the rows add up to the repository's deduplicated size. Archives sharing a name form a series, so a name usually groups all backups of one source; old-style archives that do not form a series have one name each and group just as well. This needs a single pass: flag bits 5..22 hold the name that first referenced a chunk (as index + 1, so 0 means unreferenced) and bit 23 marks chunks referenced by more than one name. As the shared and unreferenced rows require looking at every archive, --by-name always covers the whole repository and rejects archive filters. Rather than building a second index, this reuses the repository's own chunk index: it already holds every chunk's stored size (obj_size). The flag bits are OR-ed in and the source size (0 in the repo index, it is only recorded in the archives referencing a chunk) is filled in from the per-archive references cache, keeping the pack location intact via _replace. The mutations never persist: write_chunkindex_to_repo zeroes flags and size, and close() only serializes F_NEW entries, of which there are none here. Chunk membership and source sizes come from the per-archive references cache that `borg compact` maintains, so unchanged archives usually need not be opened at all. Its helpers (get_archive_references and friends, ArchiveReferences) move from archiver/compact_cmd.py to cache.py as free functions, so analyze and compact share one implementation; compact keeps its previous behaviour. The pre-existing directory hot-spot report is unchanged and still needs 2+ archives. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
borg info no longer computes per-archive deduplicated sizes (Archive.calc_stats sets usize=0 as it is expensive), and the output only shows the original size. The epilog still explained "this archive vs all archives deduplicated size", which no longer matches what the command prints. Replace it with a note that deduplicated sizes are not shown here and point to `borg analyze` (deduplicated size of a set of archives) and `borg compact --stats` (repository-wide deduplicated size). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ThomasWaldmann
force-pushed
the
analyze-set-dedup-5741
branch
from
July 30, 2026 13:19
bca673a to
28de45c
Compare
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the request in #5741: report the deduplicated size of a set of archives (e.g. all backups matching
-a 'sh:userA-*'), following ThomasWaldmann's proposal there.Two commits: the
analyzefeature, plus an unrelated one-liner fixing a staleborg infoepilog.Every figure is reported as source (uncompressed source data) and stored (compressed, as stored in the repository) size, plus the compression factor relating the two (stored / source).
Deduplicated size of a set of archives
For the considered (matching) set of archives:
borg compactcould free in the current state of the repository. Only their stored size is known: a chunk's source size is only recorded in the archives referencing it, and these have none.Following the issue comment, chunks are flagged as referenced by the considered set (C) and/or by the rest of the archives (R); the exclusive size is
sum(size(x) for x in C - R).Without an archive filter, the considered set is all archives - every referenced chunk is then trivially exclusive to it, so that line is suppressed and the report is labeled as the whole repository.
Decomposition by archive name (
--by-name)Archives sharing a name form a series, so a name usually groups all backups of one source; old-style archives that do not form a series have one name each and are grouped just as well.
Every chunk is counted in exactly one row, so the rows add up to the repository's deduplicated size. This answers "which name costs how much, and what would I get back by dropping it" for all names at once:
This needs only a single pass over the archives: bits 5..22 of the ephemeral user flags hold the archive name that first referenced a chunk (as index + 1, so 0 means unreferenced), bit 23 marks chunks referenced by more than one name. As the shared and unreferenced rows can only be determined by looking at every archive,
--by-namealways covers the whole repository and cannot be combined with archive filters.Implementation
repository.chunks) instead of building a second index — it already holds each chunk's stored size (obj_size). The membership flag bits are OR-ed in and the source size (0 in the repo index) is filled in from the per-archive references cache, preservingpack_id/obj_offset/obj_sizevia_replace. These in-memory mutations never persist:write_chunkindex_to_repozeroes flags and size, andclose()only serializesF_NEWentries (none here).borg compact, so unchanged archives usually do not need to be opened/decrypted.get_archive_references& friends,ArchiveReferences) are moved fromarchiver/compact_cmd.pyintocache.pyas free functions soanalyzeandcompactshare one implementation;compactkeeps its previous behavior.Notes
-aneeds thesh:prefix for globs (borg2's default match is exact).borg compactonly keeps them when the repository is damaged, soborg undeletestays possible).n/ain the compression column rather than dividing by zero.Tests
analyze_cmd_test.pycovers: set dedup vs exclusive size with shared and unique files, the single-archive set, the whole-repository (no filter) report, the--by-namedecomposition (asserting the rows add up to the total), the rejection of filters with--by-name, and the unreferenced-chunks report - including thatborg compactthen frees exactly the reported number of objects.Rebased onto current master; analyze/compact/cache/info suites pass there (94 tests), mypy reports no new errors.